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ABSTRACT 



This report summarizes the data obtained through the 
administration of the Observation Survey (M. Clay) as part of the K-2 Student 
Assessment Portfolio of the Durham Public Schools, North Carolina, during the 
second year of data collection. The database contained a total of 7,752 
students in kindergarten through grade 2. The Observation Survey consists of 
tasks in letter identification, words, concepts about print, writing 
vocabulary, dictation, and text reading level. Findings indicate that 
kindergarten students, in general, are mastering their letters and concepts 
about print by the end of kindergarten, although as many as one quarter may 
not have mastered their letters. In first grade, about four in five students 
were able to meet the criterion of satisfactory progress in writing 
vocabulary, and there were increases in proficiency, in comparison with the 
previous year, for dictation and text reading level. Gains for second graders 
were even greater on the text reading level task. Overall, results indicate 
that significant teaching and learning is going on in the primary grades of 
the Durham Public Schools. (Contains 10 tables.) (SLD) 



ERIC 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



TM031542 



Primary Grade Student Assessment Based 
on Tasks of the Observation Survey 



o 

o 



This paper is prepared for the: 

Annual Meeting of the American Educational Research Association in New Orleans 

April 2000 



O 



ERIC 



Joseph F. Haenn 




PERMISSION TO REPRODUCE AND 

disseminate this material has 
BEEN GRANTED BY 



lA aeJOO 



TO THE EDUCATIONAL RESOURCES 
information center (ERIC) 



1 



2 



BEST COPY AVAILABLE 



Primary Grade Student Assessment Based on 
Tasks of the Observation Survey^ 

Joseph F. Haenn 
Durham (NC) Public Schools 



Introduction 

As part of the effort by the Durham Public Schools’ K-3 Initiative^ a way was needed to obtain 
and maintain an ongoing running record of student performance during kindergarten and the 
primary grades. Due to state legislation prohibiting statewide standardized testing of students 
prior to grade 3, the North Carolina End of Grade Testing Program does not begin until grade 3. 
Therefore, Durham Public Schools had to design and implement its own assessment procedures in 
grades K-2. The result was the DPS K-2 Student Assessment Portfolio. 

A major portion of the DPS K-2 Student Assessment Portfolio is the administration of portions 
of Marie Clay’s Observation Survey at specified times during the primary school experience. The 
initial administration of the DPS K-2 Student Assessment Portfolio was completed during the 
1997-98 school year. This report summarizes the data obtained through the administration of the 
Observation Survey during the second year of data collection. 

Background of DPS Administration of the Observation Survey 

An implementation time line was developed which indicated when components of the K-2 
Student Assessment were to be administered for each quarter of each grade level, K-2. In some 
instances, all students were to be tested using a given instrument or rating scale. In other 
instances, only low performing students (designated as Low Progress students) were to be tested, 
with low performance defined as a prior performance score. Copies of the January 1999 revision 
of these time lines are provided in the appendix. 

A paper-and-pencil data collection form originally was developed to collect the information from 
each administration of a task from the Observation Survey. By mid-year of the first year of data 
collection, it was decided that these data would need to be collected and analyzed. Therefore, a 
system of data collection on diskettes using spreadsheets created in ClarisWorks^ was developed. 
A data collection diskette and backup disk were distributed to each K-2 classroom teacher at the 
beginning of the school year to allow for electronic collection of data for each of the four school 
year quarters. These diskettes were collected at the middle and again at the end of the school 
year, aggregated onto a large spreadsheet, imported into Microsoft Excels and analyzed using the 
Statistical Package for the Social Sciences (SPSS). 

The remainder of this report presents the results of the specified administrations on selected 
components of the DPS K-2 Students Assessment Portfolio related to the Observation Survey. 



* Paper presented at the Annual Meetings of the American Educational Research Association, April 2000, 
New Orleans, LA. 

^ ClarisWorks is the only systemwide spreadsheet deployed on the Macintosh computers that are located in 
all elementary schools and in many classrooms. 
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All reported data were aggregated and cleaned. Data exceeding allowable score parameters were 
treated as missing data.^ As a result of this exercise during the prior year and an effort to provide 
much better data, test administration, data entry, and data collection procedures were modified for 
the 1998-99 school year. These included designation and training of an Observation Survey 
Contact Person in each school to check diskettes before submission. Although this did not 
eliminate all problems, it did reduce the amount of cleaning necessary before data analysis. 

The resultant database consisted of 2,597 kindergarten students, 2,242 first grade students, and 
2,513 second grade students, for a total of 7,752 K-2 students. Not all students took all of the 
tasks of the Observation Survey, This report will focus on change scores and only on those 
portions of the Obsen^ation Survey that were to be administered to all students at these times. 
Separate analyses of the Low Progress students, who were tested more frequently during the 
school year, were not conducted. 

The remainder of this report presents an overview of the Observation Survey and the results of 
the end-of-year data analyses for the 1998-99 school year. Data are presented by task and grade 
level. 

Observation Survey Tasks 

The Observation Survey consists of six different tasks: Letter Identification, Word Test, Concepts 
About Print, Writing Vocabulary, Dictation, and Text Reading Level. A brief discussion of each 
of these tasks follows, although results for all of these tasks will not be provided in this report. 

Letter Identification 

The Letter Identification task assesses which letters a child knows, and which letters a child can 
identify. Letters are presented as upper case, lower case, and the typescript “a” and “g”. Scores 
range from 0 to 54. 

Word Test 

The Word Test task assesses the extent to which a child is accumulating a reading vocabulary of 
words frequently used during the first years of school. There are 20 words (with alternative lists 
for re-testing) that a child is asked to identify verbally. Scores range from 0 to 20. 

Concepts About Print 

The purpose of the Concepts About Print task is to assess what a child has learned about the way 
we print languages. Concepts include: that there are letters; that clusters of letters make words; 
that spacing between words and paragraphs has meaning; that a book goes from front to back; etc. 
Altogether there are 24 concepts, so the range of scores for students in grades 1 and 2 is 0 to 24. 
In kindergarten, only 15 of these concepts are tested, so kindergarten scores range from 0 to 15. 

Writing Vocabulary 

The fourth task— Writing Vocabulary— assesses what a child understands about print and messages 
in print as well as the features of print to which students are attending (e.g., lefl-to-right 
sequencing, letter formation). A child is allowed to write as many words as he can in 10 minutes. 
For the Portfolio assessment, a student can stop writing words after more than 60 words have 
been written. This is because only a few students at these grade levels can write that many words 



^ Allowable score ranges for each portion of the Observation Survey are: Letter Identification (0-54); Word 
Test (0-20); Concepts About Print (K=0-15; Grades l&2=0-24); Writing Vocabulary (capped at 61); 
Dictation (0-37); Test Reading Level (A,B,i-34) 
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in 10 minutes, but for those that can, they often have a quite extensive written vocabulary. Thus, 
the range of allowable scores on this task is from 0 to 61 . 

Hearing and Recording Sounds in Words (Dictation) 

This task is more commonly referred to as Dictation. It assesses a child’s ability to analyze 
spoken words (either heard or said) and to find a way to record the letters of sounds in these 
words. A child is given 1 point for each sound (phoneme) identified correctly; scores range from 
0to37. 

Text Reading Level 

This task is also known as the Running Record of Text Reading. It consists of an observation 
method for identifying and categorizing behaviors and strategies in reading books of increasing 
difficulty. Each book has an identified difficulty level. There are two pre-reading categories 
(“A” and “B”) and 34 passage levels. Thus, acceptable scores are A, B, and 1 to 34. 



Kindergarten Results 

During the fourth quarter of the 1998-99 school year, all kindergarten students were to be tested 
on two portions of the Observation Survey: Writing Vocabulary and Text Reading Level. In 
addition, all students were tested one or more times during the school year using the Letter 
Identification and Concepts About Print tasks. 

Letter Identification 



Letter Identification is a skill that needs to be mastered fairly early by students. Students need to 
know their alphabet and sounds before they can master words and reading. Therefore, a fairly 
stringent criterion was established that each student should be able to correctly identify at least 50 
of the 54 items on this task by the end of Kindergarten. The results on this task by school 
building are presented below in Table I."^ 

Overall, almost three-quarters of the students met the criterion for Letter Identification by the end 
of the school year. However, results differ markedly across school buildings ranging from highs 
of over 90 percent at Y.E. Smith and Eno Valley to lows of less than 60 percent at W.G. Pearson, 
Eastway, Little River, and Parkwood. Only 12 of the elementary scores were below the district 
average. 



^ These results underestimate the actual results obtained for the school system. The score used to 
determine whether the criterion was met for a given student was the highest score that student 
achieved at any time during the school year. Thus, if a student was present for only a portion of 
the year (i.e., came and left) but was tested before leaving on this task, then that student was 
included in this analysis. Where possible, students who transferred within the school system were 
matched based on identifying infonuation to produce a single set- of scores for that student. 
However, many students could not be matched. Thus, these percentages are the percentage of all 
students in the school system at this grade level during the school year who were able to meet the 
criterion on this task. 
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Table I 

Percent of Students At or Above Criterion on Letter Identification 
By the End of Kindergarten (By Building) 
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Concepts About Print 

Concepts About Print is another important pre-reading skill. A maximum score for kindergarten 
is 1 5, so a criterion level was set for a score of 12. The results on this task by school building are 
presented below in Table II. ^ 

Overall, almost 85 percent of the students met the criterion for Concepts About Print by the end 
of the school year. Again, results differed markedly across school buildings ranging from highs 
of over 95 percent at Y.E. Smith and Pearsontown to lows of less than 75 percent at Parkwood, 
C.C. Spaulding, and Eastway. Only 1 1 of the elementary scores were below the district average. 
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Table II 

Percent of Students At or Above Criterion on Concepts About Print 
By the End of Kindergarten (By Building) 
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Writing Vocabulary 

The Writing Vocabulary task was administered to all kindergarten students for the first time 
during the fourth quarter of the school year. Only that score is used in the following analyses. 

The average score (out of a maximum capped score of 61) for each school building is provided 
below in Table III. Overall, an average student could write 16 words. Average scores ranges 
from highs of over 20 words in Parkwood, Eno Valley, and Oak Grove to lows of less than 10 
words at Eastway, W.G. Pearson, and E.K. Powe. Only 10 of the elementary schools were above 
the district average. 
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Table III 

Mean Score of Students on Writing Vocabularj' 
By the End of Kindergarten (By Building) 




61 19.3 
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In addition to average scores, the distribution of scores within a school was also of interest. 
These distributions are presented in Table IV below. Over 40 percent of the students at W.G. 
Pearson and Eastway were able to write only 3 three words or fewer. In addition, only one-third 
of the students at Eastway were able to write more than 7 words. However, at Oak Grove and 
Parkwood almost one-quarter of the students and about 20 percent of the students at Burton and 
Eno Valley were able to write 35 or more words. At Parkwood, almost half of the students were 
able to write more than 22 words. 
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Table IV 

Percentage Distribution of Student Scores on Writing Vocabulary 
By the End of Kindergarten (By Building) 
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Text Reading Level 

The Text Reading Level indicates the approximate level of reading proficiency for each student. 
This task was administered to all kindergarten students for the first time during the fourth quarter 
of the school year. Only that score is used in the following analyses. 

The average Text Reading Level for each school building is provided below in Table V. The 
pre-reading text levels are A and B, followed by Text Reading Levels 1 through 34. Note that the 
analyses that follow may be skewed in some schools where the task was terminated as soon as a 
student reached an arbitrary level of proficiency. In other schools, students were tested until they 
reached their maximum level of proficiency. 

As expected, the overall Text Reading Level is low (i.e., 3.3). However, there is still a 
considerable variation in Text Reading Levels across buildings. These average reading levels 
range from lows at pre-reading levels for students at Bethesda and Eastway to levels that are more 
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typical of the middle of the first grade for students at R.N. Harris, Oak Grove, Morehead, 
Parkwood. Sixteen schools were reading above the district average. 



Table V 

Mean Text Readiing Level of Students 
By the End of Kindergarten (By Building) 
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In addition to average scores, the distribution of Text Reading Level scores within a school was 
of interest. These distributions are presented in Table VI below. Seven elementary schools had 
no pre-readers at the end of kindergarten, while pre-readers made up over half of the graduating 
kindergarten classes at Bethesda and more than 30 percent at E.K. Powe, George Watts, and 
Mangum. Less than 2 percent of the students at Eastway were reading above Text Reading Level 
1, and no student was reading above Text Reading Level 3. At the other end of the continuum, 
more than 20 percent of these graduating kindergarten students at Morehead and R.N. Harris were 
reading at Level 15, a level that corresponds to the 60th percentile for students at the end of the 
first grade. 
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Table VI 

Percentage Distribution of 
Student Text Reading Level Scores 
By the End of Kindergarten (By Building) 
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First Grade Results 

During both the first and fourth quarters, all first grade students were to be tested using the 
Dictation and Text Reading Level tasks. In addition, during the fourth quarter all first graders 
also were tested with the Writing Vocabulary task. Results for each of these five administrations 
are presented below. Criteria are available for acceptable performance at each of these 
administrations because normative data exists for the first grade. 

Writing Vocabulary 

The Writing Vocabulary task was to be administered to Low Progress students during each 
quarter, but to all students during the fourth quarter. The criterion for success in Writing 
Vocabulary at the end of the first grade is a score of 32 or higher. The results as to the percentage 
of students reaching this criterion in each school are presented below in Table VII. 
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Table VII 

Percent of Students At or Above Criterion on Writing Vocabulary 
By the End of Grade 1 (By Building) 




Students at most of these schools were able to meet this criterion. Over 90 percent of these 
students met the criterion at Club Boulevard, Morehead, Little River, Eno Valley, and Parkwood. 
All but 1.1 percent of the students at Club Boulevard met the criterion. However, more than one- 
quarter of the students did not meet this criterion in Writing Vocabulary at E.K. Powe, Eastway, 
Glenn, Forest View, Holt, and Lakewood. In fact, more than 1 in 3 students at E.K. Powe did not 
reach the criterion score and 1 1 of the schools were below the district average. 

Dictation 



The Dictation task was administered to a first grade students during both the first and fourth 
quarters. The criterion for not being identified as Low Performing during the first quarter was 22, 
while this criterion rose to 33 during the fourth quarter. 

As shown below in Table VIII, the overall percentage of students reaching the criterion rose by 
over 50 percent from the first to the fourth quarter even though the criterion also increased by 50 
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percent. While only about half of these first grade students met the criterion at the beginning of 
the year, more than three-quarters met the criterion by the end of the year. 



Table VIII 

Percent of Students At or Above Criterion on Dictation 
At the Beginning and End of Grade 1 (By Building) 







Burton 67 49.3% 69 63.8% 
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The percent of students reaching the criterion score during the first quarter varied widely among 
schools, from lows of only 19.2 percent at Eastway and 20,0 percent at E.K. Powe (with five 
other schools below 30 percent) to highs of 72.9 percent at Parkwood and 72.0 percent at 
Fayetteville Street. By the fourth quarter, the range was from lows of 53.5 percent at E.K. Powe, 
57.1 percent at Eastway, and 57.5 percent at Mangum to highs of 91.0 percent at Club Boulevard 
and 90.6 percent at Little River. 

The largest gains on the Dictation task during the year (in terms of percentage points) were 
demonstrated by first grade students at Y.E. Smith (55.1 points), George Watts (52.2 points). 
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R.N. Harris (48.6 points), Lakewood (47.1 points), and Bethesda (45.6 points). However, it must 
be emphasized that these gains are not based on matched pairs of students. 

Text Reading Level 

Text Reading Levels were gathered from all students in the first grade during both the first and 
fourth quarters. The Text Reading Level criterion for not being identified as Low Performing 
during the first quarter was 2, while this criterion rose to 9 during the fourth quarter. 

As shown below in Table IX, the overall percentage of students reaching the criterion rose by 
almost 20 percent from the first to the fourth quarter even though the criterion more than 
quadrupled. While only about two-thirds of these first grade students met the criterion at the 
beginning of the year, more than 80 percent met the criterion by the end of the year. 



Table IX 

Percent of Students At or Above Criterion on Text Reading Level 




At the Beginning and End of Grade 1 (By Building) 
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The percentage of students reaching the criterion score during the first quarter varied from lows 
of less than 42 percent at W.G. Pearson, Eastway, and George Watts (with three other schools 
below 50 percent) to highs of over 85 percent at Y.E. Smith, Southwest, and Club Boulevard. By 
the fourth quarter, the lower end of the range of scores had increased to over 63 percent at every 
school except E.K. Powe. Meanwhile, the highs increased to over 95 percent at Easley and Club 
Boulevard and over 90 percent at Southwest, Fayetteville Street, and Little River. 

The largest gains on Text Reading Level for first grade students during the year (in terms of 
percentage. points) were demonstrated by students at George Watts (43.7 points), W.G. Pearson 
(30.1 points), and Burton and Morehead (each 29.2 points). However, it must be emphasized that 
these gains are not based on matched pairs of students. 



Second Grade Results 

There were no normative data available for second grade students. However, all second graders 
were to be administered the Text Reading Level task during both the first and fourth quarters. 
Therefore, growth on this measure relative to criterion scores can be examined. The Text 
Reading Level criterion for not being identified as Low Performing during the first quarter was 
13, while this criterion rose to 17 during the fourth quarter. 

As shown below in Table X, the overall percentage of students reaching the criterion rose by 
about 23 percent from the first to the fourth quarter while the criterion was increased by about 30 
percent. While over 70 percent of these second grade, students met the criterion at the beginning 
. of the year, almost 90 percent met the criterion by the end of the year. 

The percentage of students reaching the criterion score during the first quarter varied from lows 
of less than 50 percent at C.C. Spaulding and George Watts (with five other schools below 55 
percent) to highs of over 85 percent at Club Boulevard, Easley, and Mangum. By the fourth 
quarter, the lower end of the range of scores had increased to over 70 percent at every school 
except C.C. Spaulding. Meanwhile, the highs increased to over 95 percent at six schools, 
including almost unbelievable rates of 99.1 percent at Easley and 98.7 percent at Mangum. 

The largest gains on Text Reading Level for second grade students during the year (in terms of 
percentage points) were demonstrated by students at Bethesda (36.3 points), R.M. Harris (34.4 
point), Y.E. Smith (29.6 points), and Burton (28.1 points). However, it must be emphasized that 
these gains are not based on matched pairs of students. 



Conclusions 

Kindergarten students, in general, are mastering their letters (Letter Identification) and Concepts 
About Print by the end of kindergarten. However, results indicate the possibility that as many as 
one-quarter of these students may not have mastered their letters and over 15 percent may n^ 
have mastered concepts about print by the end of kindergarten. This finding encouraged the 
evaluator to examine individual student data more closely. His finding was that about seven 
percent of the students that were analyzed on the Letter Identification task were not re-tested in 
the fourth quarter as called for in the guidelines. Some of these students might have left the 
classroom (e.g., moved, transferred), but others met the criterion for an earlier quarter and simply 
were not tested again during the fourth quarter. Also in kindergarten, average mastery of Writing 
Vocabulary and Text Reading Level fluctuates quite a bit among the elementary schools. For 
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Writing Vocabulary, three schools averaged having kindergarten students with writing 
vocabularies of less than 10 words, and three schools had averages of more than 22 words. 
Average Text Reading Level ranged from pre-readers (in two schools) to an average Text 
Reading Level of more than 6.8 (in two other schools). 



Table X 

Percent of Students At or Above Criterion on Text Reading Level 
At the Beginning and End of Grade 2 (By Building) 







Burton 64 53.1% 69 81.2% 
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George Watts 60 48.3% 63 71.4% 
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In grade 1, about 4 in 5 of the students were able to meet the criterion of satisfactory progress 
(i.e., not deemed as Low Progress) in Writing Vocabulary. However, these figures included 
almost all students in one school but less than 80 percent of the students in 1 1 other schools. 
There was a marked increase in proficiency (defined as percentage of student meeting a criterion) 
in both Dictation and Text Reading Level for first grade students. On the Dictation task, there 
was better than a 50 percent increase in proficiency, with two schools demonstrating over a 50 
percentage point increase in proficiency. On the Text Reading Level task, there was almost a 20 
percent increase in proficiency, with one school demonstrating over a 40 percentage point 
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increase in proficiency. However, schools were identified that are not getting nearly enough 
students to a level of proficiency in both of these areas. 

For second graders, gains were even greater on the Text Reading Level task. Three schools 
demonstrated 30 percentage point increases in Text Reading Level. However, at a time when 
almost all students should be reading at a satisfactory level, these levels are still too low (i.e., less 
than 80 percent proficient) in six schools. 

These results indicate that significant teaching and learning is occurring in the primary grades in 
Durham Public Schools. However, they also pinpoint areas of weakness that need to be 
addressed so that all student learn to read adequately by the third grade. 
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